[BETA] Automated PR review — SeedSource AI pipeline

Advisory only, not a merge gate. AI-generated (internal risk-triage, Atlas serve gate, GLM-5.2 review), attributed per Atlas convention.

Triage

• Risk: med - serve-test-required (triage advisory): True - path-heuristic (authority): True
• Rationale: Adds n>=4 NVFP4 tiled GEMV paths for o_proj and qkv projections using proven batch kernels.
• Serving-path files: crates/spark-model/src/layers/qwen3_attention/trait_impl/multi_seq/attn.rs, crates/spark-model/src/layers/qwen3_attention/trait_impl/multi_seq/qkv.rs

Serve gate

• Build: 18s - Qwen3.6-35B-A3B-FP8 on GB10 node1
• Top-k KL vs main: 0.0006 (no shift vs main) - decode delta +0.1%
• Coherence probe: coherence=pass ping=10/10 (note: our test_coherence.py is flaky here; KL is the reliable serve signal)

GLM-5.2 code review

Correctness concerns:

1. Scratch buffer aliasing (HIGH): q_scratch = ssm_qkvz(), k_scratch = attn_output(). v_scratch = k_scratch.offset(k * kv_bytes). If attn_output buffer isn't sized for k*kv_bytes + k*kv_bytes (k+v scratch), this OOB-writes. The batch3/batch2 paths presumably validated this, but tiling reuses the same buffers per tile — fine if sized for max tile (3), but no assertion guards it.

2. o_proj branch condition: else if !self.attn.o_proj.is_null() — this catches n>=4 AND n==1 (falls through from the n==2/3 branches above). For n==1 this tiles into a single k==1 GEMV call, which is functionally correct but a pointless wrapper vs. the existing else fallback. Verify the else branch is now truly dead for NVFP4 o_proj or that n==1 still hits it.

3. D2D copies not batched: 3 separate copy_d2d_async per token in the scatter loop (9 for k==3 tile). These are on the same stream so serialized; minor perf but not a correctness issue.

4. No synchronization between tiles: Tiles reuse the same scratch buffers with no stream sync between iterations. Correct only if all kernels are enqueued on the same stream (appears so), but worth confirming no cross-stream dependencies exist.

Risk: Medium. Buffer sizing is the critical unverified assumption.

Quality: Tiling logic is clean; the k==1 remainder fallback duplicating the sequential path is acceptable but could call into a shared helper to avoid drift.

_{Pipeline: build+serve on GB10, GLM-5.2 via NVIDIA NIM, internal risk-triage classifier. External diff treated as untrusted input. Public repo.}

GPU A/B — tiled-GEMV (this PR) vs wide-GEMM (#305), on current main

Fulfilling the "GPU numeric A/B recommended before merge" item. Since #305 (DFlash commit_ctx) landed on main, the n≥4 o_proj/QKV branch this PR targets is now occupied by #305's wide_verify_gemm (single M=n GEMM) — so this is a direct head-to-head. Both variants built on the identical current-main base, differing only in the n≥4 kernel.

Model: AEON-7/Ornith-1.0-35B-AEON-…-NVFP4 (uniform NVFP4 attn) · GB10/sm_121 · temp 0 · 256-token decode-dominant · bf16 KV · max-num-seqs 16. Concurrency = simultaneous identical requests (batched decode). per-seq = server-reported response_token/s; agg = wall-clock aggregate.

Result: the tiled batch3/batch2 GEMV beats the wide GEMM by ~1.5–1.7× at every batched size — opposite to the "one weight read wins" intuition. The wide GEMM's M-tile padding (94% waste at M=4) outweighs its single-read advantage; note wide's C=4 aggregate (59.9) is actually below its C=1 (89.2) — it degrades under moderate batch, which the tiling fixes. Even at C=16 (≈ #305's γ=16 target) tiled wins.

Resolution in this sync

This branch is now rebased onto current main with the n≥4 conflict resolved toward tiled — i.e. it supersedes wide_verify_gemm for that branch (the function is removed; the n==2/3 batch kernels are untouched). Justified by the table above.

Caveats / remaining before merge

• Single model / prompt / greedy / one run. Directionally very strong (and since this is a MoE+SSM model where attention is only part of decode, the raw attention-kernel win is likely larger than the end-to-end 1.5×), but a multi-model sweep would firm up the exact factor.
• Numeric identity: the outstanding tiled-vs-per-seq output-identity check (this PR's own correctness argument) is still the must-do — temp-0 output diverges from the wide kernel (expected: different accumulation order; both coherent/correct), so it must be pinned against the n==1 per-seq reference, not against wide.
• DFlash interaction: wide_verify_gemm was introduced by feat(dflash): unified context commit (commit_ctx), env-gated (stacked on #304) #305 for the DFlash γ=16 verify step, which flows through this same n≥4 branch. Removing it routes DFlash verify through the tiled path too. That's functionally fine (tiled handles any n≥4) and faster here, but a --dflash-on run should confirm no verify-path regression before this lands, since feat(dflash): unified context commit (commit_ctx), env-gated (stacked on #304) #305's kernel was tuned for that specific call site.

DFlash verify-path check — no regression from removing wide_verify_gemm

Closing the third caveat from the A/B above. wide_verify_gemm was introduced by #305 for the DFlash γ=16 verify step; this branch removes it and routes verify (an n=γ+1=16 batch) through the tiled path. Ran both binaries with --dflash (Ornith-1.0-35B-NVFP4 target + z-lab/Qwen3.6-35B-A3B-DFlash drafter, full DFlash correctness flags, γ=16):

• Verify-step timing identical within noise (~87 ms). feat(dflash): unified context commit (commit_ctx), env-gated (stacked on #304) #305's wide GEMM shows no advantage even in its own DFlash-verify context — the verify step is dominated by drafter-propose + SSM/MoE forward, so the o_proj/QKV kernel washes out there (unlike concurrent batched decode, where tiled is 1.5×).
• Accept rate is kernel-independent (identical math, ~5–13% — that low rate is the known 35B-A3B/drafter divergence, unrelated to this PR).
• Both coherent, no illegal-address / crash through the tiled verify batch.

Conclusion: removing wide_verify_gemm and using tiled everywhere is safe — faster for concurrent decode, equal for DFlash verify, no downside. This PR is synced onto current main with that resolution. Remaining pre-merge item is the tiled-vs-n==1-per-seq numeric-identity check.

High-acceptance verify test — tiled still wins

The DFlash run above had low accept (~5%, adaptive suspended), so the verify path barely ran. To stress a productive n≥4 verify, ran MTP forced past its throughput gate (ATLAS_MTP_GATE_FORCE=1, --num-drafts 3 → K=4, verify batch n=4) on unsloth/Qwen3.6-35B-A3B-NVFP4 (the config that gets high spec-accept). Both binaries, identical flags:

Tiled is ~1.18× faster end-to-end even with the verify running every step at high acceptance, and accept is at parity (the 1.72↔1.93 spread is temp-0 numeric drift between the two accumulation kernels over a single run, not a systematic edge). Both coherent, no crash.

Verify-path summary across regimes:

• Concurrent batched decode (C=4/8/16): tiled 1.49–1.69×.
• DFlash verify (n=16, low accept, verify-dominated-by-forward): tiled ≈ wide (no regression).
• MTP verify (n=4, ~60% accept, productive): tiled ~1.18×.

Net: tiled is faster or equal in every regime tested — concurrent decode, low-accept DFlash verify, and high-accept MTP verify. Removing wide_verify_gemm in favor of tiled carries no downside on the evidence here. Remaining pre-merge item stays the tiled-vs-n==1-per-seq numeric-identity check.

Second model (Holo-3.1) — the win is model-dependent

Ran the same n≥4 concurrent-decode sweep on Hcompany/Holo-3.1-35B-A3B-NVFP4 (qwen3_5_moe). (MTP-force is N/A here — Holo ships no MTP head: 0 MTP weight keys, mtp_num_hidden_layers: None.)

On Holo the tiled advantage is marginal (1.0–1.10×) — not the 1.49–1.69× seen on Ornith. The cause is in the wide-GEMM baseline: on Ornith (qwen3_6_moe) wide degraded at moderate batch (C=4 aggregate below C=1) and tiling recovered it; on Holo (qwen3_5_moe) wide already scales cleanly (C=4 agg 104.6 vs C=1 76.6), leaving little for tiling to gain.

Revised takeaway

The tiled kernel win is model/shape-dependent, not universal: large on Ornith, near-parity on Holo, and never a regression across everything tested (concurrent decode on 2 models, low-accept DFlash verify, high-accept MTP verify). So tiled is a safe drop-in — it helps some models substantially and costs nothing elsewhere — but the honest headline is "up to ~1.7×, ≥ parity everywhere," not a flat 1.5×. Whether that justifies carrying the tiled path over #305's wide GEMM is a maintainer call; the data says there's no downside to doing so.

Tiebreaker: the win is the BF16→FP4 requant path, NOT model/shape

Ran the A/B on AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4 — same qwen3_6 arch and byte-identical attention shape as Ornith (16 heads / 2 KV / head_dim 256 / hidden 2048), but native NVFP4-packed attention on disk instead of Ornith's BF16 (which gets runtime-requantized to FP4 under ATLAS_HOLO_FP4_PROJ_DECODE=1).

Full heretic sweep: C=4 1.07×, C=8 1.06×, C=16 1.01× — i.e. near-parity, like Holo, not Ornith's 1.5–1.7×.

Conclusion: the earlier "model/shape-dependent" framing was wrong. Same shape, opposite result. The discriminator is the attention weight source:

• Native NVFP4/FP8 attention (heretic, Holo, unsloth, etc.): wide-GEMM is already fast → tiled ≈ parity.
• BF16 attention runtime-requantized to FP4 (Ornith): wide-GEMM is pathological (its fast pipelined m128_v2 needs a transposed weight copy the requant path apparently doesn't build, so it falls to the slow M64 GEMM). Tiling — which uses the transpose-free batch3/batch2 GEMV — sidesteps that and wins ~1.7×.

Implication for the PR: the tiled kernel is only a meaningful win for runtime-requantized-BF16 attention checkpoints (a niche: e.g. the AEON "Ultimate" BF16 repacks). For the mainstream native-NVFP4/FP8 releases it's parity. That narrows the value proposition considerably — and suggests the alternative fix worth weighing is building the transposed NVFP4 copy during the BF16→FP4 requant (so wide_verify_gemm's fast path works too), rather than carrying a second kernel. Either way, tiled remains ≥ parity everywhere, so it's a safe merge — but it's not the broad speedup the Ornith-only numbers suggested.

Follow-up: keeping Ornith's attention in native BF16 doesn't help either

Since the Ornith win traced to the BF16→FP4 requant (not shape), the obvious question was: just don't requant — drop ATLAS_HOLO_FP4_PROJ_DECODE and serve Ornith's attention in its native BF16. Result: no better.

Ornith per-seq, all attention paths (Qwen3.6-35B-A3B, C = concurrent decode):

Native BF16 lands at exactly 15.8 — same as the pathological wide path. BF16 attention has no fast batched kernel at n≥4 either; it falls to the per-token seq-loop (n full-weight GEMVs at 2 bytes/weight), which is just as slow as the transpose-less wide-GEMM and forfeits the FP4 bandwidth savings. So dropping the requant is a net loss, not a fix.

Complete picture for the n≥4 attention path

• Native NVFP4/FP8 attention (heretic, Holo, unsloth, mainstream releases): wide already fast → tiled ≈ parity (1.0–1.1×).
• BF16-repack attention runtime-requantized to FP4 (Ornith-class "Ultimate" repacks): only three options at n≥4 — BF16 seq-loop (15.8), requant+wide (15.8), or requant+tiled (26.7). Tiling is the only one that's fast.

Bottom line for merge: tiled is ≥ parity everywhere tested (2 native-quant models, DFlash verify, MTP verify) and the only fast option for BF16-repack checkpoints at concurrency. It's a safe drop-in with a real (if niche) benefit. The equally-valid alternative — emit the transposed NVFP4 copy during the BF16→FP4 requant so wide_verify_gemm's fast path fires (matching heretic's native 27.6) — would remove the need for a second kernel; worth weighing, but out of scope for this PR.